I am very interested in this. Let's hope it works and we can move on from julia 1.10

I am disabling the reactant tests for the moment to check if the rest works.

If docs still break on the internal_tide.jl example NaNing, I would try with a GridFittedBottom instead of a PartialCellBottom, which is not even tested, and, if tests pass, merge this PR and move forward trying to assess what the problems with partial cells are.

Seems that we are hitting the same NaN issue on the internal tide example

Seems that we are hitting the same NaN issue on the internal tide example

the ghosts of the past still haunt us....

Apparently also GridFittedBottom fails...

If I run the example locally, it works. Why would it error on CI? Do we have a way to reproduce this error locally?

If I run the example locally, it works. Why would it error on CI? Do we have a way to reproduce this error locally?

One thing to try might be to run the example locally and on CI using the exact same Manifest.toml if possible. We can commit a Manifest.toml to this branch for debugging. I can't think of which dependency would lead to such a big difference but it's one thing we can control for.

From the Julia v1.11 chat I recall that the error was showing up only for unix, not for mac?

The plot thickens: it works correctly in Julia v1.12 on Ampere eMAG (aarch64) with AlmaLinux 8.10 as operating system, which rules out an operating system difference. aarch64 is also the architecture on macOS, so I'm starting to suspect there's an architecture dependence. Can someone point me to the operation performed on the u field? Is there any chance there you're running any BLAS operation? Edit: using in x86-64 Julia v1.12 the libopenblas shipped in Julia v1.10 doesn't fix the issue, so that'd rule out BLAS version differences. Edit 2: also works on Intel(R) Xeon(R) CPU X5670 with CentOS 8 Stream, I'm getting more and more confused, sigh. Pointers to the relevant code would still be very welcome.

The plot thickens: it works correctly in Julia v1.12 on Ampere eMAG (aarch64) with AlmaLinux 8.10 as operating system, which rules out an operating system difference. aarch64 is also the architecture on macOS, so I'm starting to suspect there's an architecture dependence. Can someone point me to the operation performed on the u field? Is there any chance there you're running any BLAS operation? Edit: using in x86-64 Julia v1.12 the libopenblas shipped in Julia v1.10 doesn't fix the issue, so that'd rule out BLAS version differences. Edit 2: also works on Intel(R) Xeon(R) CPU X5670 with CentOS 8 Stream, I'm getting more and more confused, sigh. Pointers to the relevant code would still be very welcome.

Nice work so far though!!

The entire time-step is a complex chain of operations. I do think it is a good start to save down all fields every time-step. We may find that differences arise in one field versus another. Note that the NaNChecker checks u only as a proxy for the entire state. Here the prognostic state should be model.velocities (u, v, w) and model.tracers.b.

To save every iteration chnage this line

to schedule = IterationInterval(1). Also I think we should add v (I see u and w but not v there).

The difference should arise in the very first time-step? We could compare those. It seems annoying laborious to do this across architectures, but maybe @giordano you have good ideas how to do this efficiently

Shall we just move the tests to julia 1.11 then until Reactant and Enzyme are ready?

How easy would it be to run Reactant/Enzyme tests only with 1.11 in Buildkite? It shouldn't be hard in github actions, but I'm less familiar with buildkite.

I can try giving it a go. We can actually add another pipeline, and probably need a new queue. @ali-ramadhan would you know how to set this up on nautilus if I separate out the tests in a new pipeline?

I think I have managed to separate them out.

Alright, I believe I found the right incantation in 94bc181.

The summary is

• the step 🏕️ initialize general environment was instantiating the first environment with Julia v1.12, creating the Manifest.toml
• the step 🏕️ initialize enzyme environment would try to use the Manifest.toml generated by the previous step (the two steps happen one after the other, in the same machine & directory) with Julia v1.11, failing.

My solution was to

• rename the manifest generated by the first step to the version-specific name after instantiation
• also rename the Enzyme manifest to the version-specific name after instantiation. This is strictly unnecessary if the previous step is already disambiguating the manifests, but having version-specific names everywhere is a more robust approach anyway.

The docs built and the internal tide example looks great!
https://clima.github.io/OceananigansDocumentation/previews/PR4836/literated/internal_tide/

There seem to be some errors in the GPU abstract operations CI:

https://buildkite.com/clima/oceananigans/builds/26558#019a515f-fc97-4a52-96e9-a8a5894dffdd/7-53

If that smooths out, I'll work on bringing the doctests back in!